{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text"
   },
   "source": [
    "# Image classification from scratch\n",
    "\n",
    "**Author:** [fchollet](https://twitter.com/fchollet)<br>\n",
    "**Date created:** 2020/04/27<br>\n",
    "**Last modified:** 2020/04/28<br>\n",
    "**Description:** Training an image classifier from scratch on the Kaggle Cats vs Dogs dataset."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text"
   },
   "source": [
    "## Introduction\n",
    "\n",
    "This example shows how to do image classification from scratch, starting from JPEG\n",
    "image files on disk, without leveraging pre-trained weights or a pre-made Keras\n",
    "Application model. We demonstrate the workflow on the Kaggle Cats vs Dogs binary\n",
    " classification dataset.\n",
    "\n",
    "We use the `image_dataset_from_directory` utility to generate the datasets, and\n",
    "we use Keras image preprocessing layers for image standardization and data augmentation.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text"
   },
   "source": [
    "## Setup\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab_type": "code"
   },
   "outputs": [],
   "source": [
    "import tensorflow as tf\n",
    "from tensorflow import keras\n",
    "from tensorflow.keras import layers\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text"
   },
   "source": [
    "## Load the data: the Cats vs Dogs dataset\n",
    "\n",
    "### Raw data download\n",
    "\n",
    "First, let's download the 786M ZIP archive of the raw data:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab_type": "code"
   },
   "outputs": [],
   "source": [
    "!curl -O https://download.microsoft.com/download/3/E/1/3E1C3F21-ECDB-4869-8368-6DEBA77B919F/kagglecatsanddogs_3367a.zip\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab_type": "code"
   },
   "outputs": [],
   "source": [
    "!unzip -q kagglecatsanddogs_3367a.zip\n",
    "!ls\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text"
   },
   "source": [
    "Now we have a `PetImages` folder which contain two subfolders, `Cat` and `Dog`. Each\n",
    " subfolder contains image files for each category.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab_type": "code"
   },
   "outputs": [],
   "source": [
    "!ls PetImages\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text"
   },
   "source": [
    "### Filter out corrupted images\n",
    "\n",
    "When working with lots of real-world image data, corrupted images are a common\n",
    "occurence. Let's filter out badly-encoded images that do not feature the string \"JFIF\"\n",
    " in their header.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab_type": "code"
   },
   "outputs": [],
   "source": [
    "import os\n",
    "\n",
    "num_skipped = 0\n",
    "for folder_name in (\"Cat\", \"Dog\"):\n",
    "    folder_path = os.path.join(\"PetImages\", folder_name)\n",
    "    for fname in os.listdir(folder_path):\n",
    "        fpath = os.path.join(folder_path, fname)\n",
    "        try:\n",
    "            fobj = open(fpath, \"rb\")\n",
    "            is_jfif = tf.compat.as_bytes(\"JFIF\") in fobj.peek(10)\n",
    "        finally:\n",
    "            fobj.close()\n",
    "\n",
    "        if not is_jfif:\n",
    "            num_skipped += 1\n",
    "            # Delete corrupted image\n",
    "            os.remove(fpath)\n",
    "\n",
    "print(\"Deleted %d images\" % num_skipped)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text"
   },
   "source": [
    "## Generate a `Dataset`\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab_type": "code"
   },
   "outputs": [],
   "source": [
    "image_size = (180, 180)\n",
    "batch_size = 32\n",
    "\n",
    "train_ds = tf.keras.preprocessing.image_dataset_from_directory(\n",
    "    \"PetImages\",\n",
    "    validation_split=0.2,\n",
    "    subset=\"training\",\n",
    "    seed=1337,\n",
    "    image_size=image_size,\n",
    "    batch_size=batch_size,\n",
    ")\n",
    "val_ds = tf.keras.preprocessing.image_dataset_from_directory(\n",
    "    \"PetImages\",\n",
    "    validation_split=0.2,\n",
    "    subset=\"validation\",\n",
    "    seed=1337,\n",
    "    image_size=image_size,\n",
    "    batch_size=batch_size,\n",
    ")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text"
   },
   "source": [
    "## Visualize the data\n",
    "\n",
    "Here are the first 9 images in the training dataset. As you can see, label 1 is \"dog\"\n",
    " and label 0 is \"cat\".\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab_type": "code"
   },
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "\n",
    "plt.figure(figsize=(10, 10))\n",
    "for images, labels in train_ds.take(1):\n",
    "    for i in range(9):\n",
    "        ax = plt.subplot(3, 3, i + 1)\n",
    "        plt.imshow(images[i].numpy().astype(\"uint8\"))\n",
    "        plt.title(int(labels[i]))\n",
    "        plt.axis(\"off\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text"
   },
   "source": [
    "## Using image data augmentation\n",
    "\n",
    "When you don't have a large image dataset, it's a good practice to artificially\n",
    "introduce sample diversity by applying random yet realistic transformations to the\n",
    "training images, such as random horizontal flipping or small random rotations. This\n",
    "helps expose the model to different aspects of the training data while slowing down\n",
    " overfitting.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab_type": "code"
   },
   "outputs": [],
   "source": [
    "data_augmentation = keras.Sequential(\n",
    "    [\n",
    "        layers.experimental.preprocessing.RandomFlip(\"horizontal\"),\n",
    "        layers.experimental.preprocessing.RandomRotation(0.1),\n",
    "    ]\n",
    ")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text"
   },
   "source": [
    "Let's visualize what the augmented samples look like, by applying `data_augmentation`\n",
    " repeatedly to the first image in the dataset:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab_type": "code"
   },
   "outputs": [],
   "source": [
    "plt.figure(figsize=(10, 10))\n",
    "for images, _ in train_ds.take(1):\n",
    "    for i in range(9):\n",
    "        augmented_images = data_augmentation(images)\n",
    "        ax = plt.subplot(3, 3, i + 1)\n",
    "        plt.imshow(augmented_images[0].numpy().astype(\"uint8\"))\n",
    "        plt.axis(\"off\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text"
   },
   "source": [
    "## Standardizing the data\n",
    "\n",
    "Our image are already in a standard size (180x180), as they are being yielded as\n",
    "contiguous `float32` batches by our dataset. However, their RGB channel values are in\n",
    " the `[0, 255]` range. This is not ideal for a neural network;\n",
    "in general you should seek to make your input values small. Here, we will\n",
    "standardize values to be in the `[0, 1]` by using a `Rescaling` layer at the start of\n",
    " our model.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text"
   },
   "source": [
    "## Two options to preprocess the data\n",
    "\n",
    "There are two ways you could be using the `data_augmentation` preprocessor:\n",
    "\n",
    "**Option 1: Make it part of the model**, like this:\n",
    "\n",
    "```python\n",
    "inputs = keras.Input(shape=input_shape)\n",
    "x = data_augmentation(inputs)\n",
    "x = layers.experimental.preprocessing.Rescaling(1./255)(x)\n",
    "...  # Rest of the model\n",
    "```\n",
    "\n",
    "With this option, your data augmentation will happen *on device*, synchronously\n",
    "with the rest of the model execution, meaning that it will benefit from GPU\n",
    " acceleration.\n",
    "\n",
    "Note that data augmentation is inactive at test time, so the input samples will only be\n",
    " augmented during `fit()`, not when calling `evaluate()` or `predict()`.\n",
    "\n",
    "If you're training on GPU, this is the better option.\n",
    "\n",
    "**Option 2: apply it to the dataset**, so as to obtain a dataset that yields batches of\n",
    " augmented images, like this:\n",
    "\n",
    "```python\n",
    "augmented_train_ds = train_ds.map(\n",
    "  lambda x, y: (data_augmentation(x, training=True), y))\n",
    "```\n",
    "\n",
    "With this option, your data augmentation will happen **on CPU**, asynchrously, and will\n",
    " be buffered before going into the model.\n",
    "\n",
    "If you're training on CPU, this is the better option, since it makes data augmentation\n",
    " asynchronous and non-blocking.\n",
    "\n",
    "In our case, we'll go with the first option.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text"
   },
   "source": [
    "## Configure the dataset for performance\n",
    "\n",
    "Let's make sure to use buffered prefetching so we can yield data from disk without\n",
    " having I/O becoming blocking:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab_type": "code"
   },
   "outputs": [],
   "source": [
    "train_ds = train_ds.prefetch(buffer_size=32)\n",
    "val_ds = val_ds.prefetch(buffer_size=32)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text"
   },
   "source": [
    "## Build a model\n",
    "\n",
    "We'll build a small version of the Xception network. We haven't particularly tried to\n",
    "optimize the architecture; if you want to do a systematic search for the best model\n",
    " configuration, consider using\n",
    "[Keras Tuner](https://github.com/keras-team/keras-tuner).\n",
    "\n",
    "Note that:\n",
    "\n",
    "- We start the model with the `data_augmentation` preprocessor, followed by a\n",
    " `Rescaling` layer.\n",
    "- We include a `Dropout` layer before the final classification layer.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab_type": "code"
   },
   "outputs": [],
   "source": [
    "\n",
    "def make_model(input_shape, num_classes):\n",
    "    inputs = keras.Input(shape=input_shape)\n",
    "    # Image augmentation block\n",
    "    x = data_augmentation(inputs)\n",
    "\n",
    "    # Entry block\n",
    "    x = layers.experimental.preprocessing.Rescaling(1.0 / 255)(x)\n",
    "    x = layers.Conv2D(32, 3, strides=2, padding=\"same\")(x)\n",
    "    x = layers.BatchNormalization()(x)\n",
    "    x = layers.Activation(\"relu\")(x)\n",
    "\n",
    "    x = layers.Conv2D(64, 3, padding=\"same\")(x)\n",
    "    x = layers.BatchNormalization()(x)\n",
    "    x = layers.Activation(\"relu\")(x)\n",
    "\n",
    "    previous_block_activation = x  # Set aside residual\n",
    "\n",
    "    for size in [128, 256, 512, 728]:\n",
    "        x = layers.Activation(\"relu\")(x)\n",
    "        x = layers.SeparableConv2D(size, 3, padding=\"same\")(x)\n",
    "        x = layers.BatchNormalization()(x)\n",
    "\n",
    "        x = layers.Activation(\"relu\")(x)\n",
    "        x = layers.SeparableConv2D(size, 3, padding=\"same\")(x)\n",
    "        x = layers.BatchNormalization()(x)\n",
    "\n",
    "        x = layers.MaxPooling2D(3, strides=2, padding=\"same\")(x)\n",
    "\n",
    "        # Project residual\n",
    "        residual = layers.Conv2D(size, 1, strides=2, padding=\"same\")(\n",
    "            previous_block_activation\n",
    "        )\n",
    "        x = layers.add([x, residual])  # Add back residual\n",
    "        previous_block_activation = x  # Set aside next residual\n",
    "\n",
    "    x = layers.SeparableConv2D(1024, 3, padding=\"same\")(x)\n",
    "    x = layers.BatchNormalization()(x)\n",
    "    x = layers.Activation(\"relu\")(x)\n",
    "\n",
    "    x = layers.GlobalAveragePooling2D()(x)\n",
    "    if num_classes == 2:\n",
    "        activation = \"sigmoid\"\n",
    "        units = 1\n",
    "    else:\n",
    "        activation = \"softmax\"\n",
    "        units = num_classes\n",
    "\n",
    "    x = layers.Dropout(0.5)(x)\n",
    "    outputs = layers.Dense(units, activation=activation)(x)\n",
    "    return keras.Model(inputs, outputs)\n",
    "\n",
    "\n",
    "model = make_model(input_shape=image_size + (3,), num_classes=2)\n",
    "keras.utils.plot_model(model, show_shapes=True)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text"
   },
   "source": [
    "## Train the model\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab_type": "code"
   },
   "outputs": [],
   "source": [
    "epochs = 50\n",
    "\n",
    "callbacks = [\n",
    "    keras.callbacks.ModelCheckpoint(\"save_at_{epoch}.h5\"),\n",
    "]\n",
    "model.compile(\n",
    "    optimizer=keras.optimizers.Adam(1e-3),\n",
    "    loss=\"binary_crossentropy\",\n",
    "    metrics=[\"accuracy\"],\n",
    ")\n",
    "model.fit(\n",
    "    train_ds, epochs=epochs, callbacks=callbacks, validation_data=val_ds,\n",
    ")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text"
   },
   "source": [
    "We get to ~96% validation accuracy after training for 50 epochs on the full dataset.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text"
   },
   "source": [
    "## Run inference on new data\n",
    "\n",
    "Note that data augmentation and dropout are inactive at inference time.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab_type": "code"
   },
   "outputs": [],
   "source": [
    "img = keras.preprocessing.image.load_img(\n",
    "    \"PetImages/Cat/6779.jpg\", target_size=image_size\n",
    ")\n",
    "img_array = keras.preprocessing.image.img_to_array(img)\n",
    "img_array = tf.expand_dims(img_array, 0)  # Create batch axis\n",
    "\n",
    "predictions = model.predict(img_array)\n",
    "score = predictions[0]\n",
    "print(\n",
    "    \"This image is %.2f percent cat and %.2f percent dog.\"\n",
    "    % (100 * (1 - score), 100 * score)\n",
    ")\n"
   ]
  }
 ],
 "metadata": {
  "colab": {
   "collapsed_sections": [],
   "name": "image_classification_from_scratch",
   "private_outputs": false,
   "provenance": [],
   "toc_visible": true
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}